Reproducible research

Encouraging code sharing in academia

Stephen J Eglen                  Cambridge Computational Biology Institute
https://sje30.github.io          University of Cambridge
sje30@cam.ac.uk                  @StephenEglen

Slides: http://bit.ly/eglen-futurepub

Acknowledgements

Co-authors, Freeman lab, Laurent Gatto.

These slides are available under a creative common CC-BY license.

Inverse problems are hard

Research sharing: the inverse problem

Where is the scholarship?

An article about computational science in a scientific publication is not the scholarship itself, it is merely advertising of the scholarship. The actual scholarship is the complete software development environment and that complete set of instructions that generated the figures.

[Buckheit and Donoho 1995, after Claerbout]

Moral or selfish approach?

Selfish reasons to share

Why not align what is good for science with what is good for scientists?

1. Funding mandates (REF + enforcement from Wellcome Trust)
2. Credit through data papers
3. Leads to further collaborations (e.g. “EPAmeadev”)
4. Fixes data bugs / errors in analysis
5. Prevent data loss (Vines et al 2014). e.g. students have a habit of leaving…
6. Your future self is probably one of the main beneficiaries of sharing.
7. Now is a very good time to be an open scientist.

Code sharing: a way forward

Specific recommendations

1. Include enough code to reproduce key figure/result from your paper (“modeldb”).
2. Provide toy examples if your project is too intensive to expect others to run in a few hours.
3. Version control (github)
4. Licence (MIT)
5. Provide data
6. Provide tests
7. Use standards
8. Use permanent URLs (Zenodo/figshare)

Simple example

Docker

Can bundle entire open-source evironment for others to share:

(start docker)
docker run -d -p 8787:8787 sje30/eglen2015
open http://192.168.99.100:8787/

This should launch a web page …

A dirty secret

• Docker does not ensure your code will work forever…
• My reproducible review lasted about 6 months!
• Continuous Intgregation would have helped find when it broke.
• Titus Brown reports similar: How I learned to stop worrying and love the coming archivability crisis in scientific software

Jupyter notebooks

• Embed code within manuscript; figures/tables dynamically regenerated

binder = Docker + jupyter + cloud compute

• mybinder.org developed and supported by Freeman lab, Janelia Farm.
• Allows jupyter notebooks to be dynamically evaluated (not just rendered) online.

binder other examples

Small example github repo

LIGO experiments

Find a code buddy

• We ask our students to submit a .Rnw file rather than a pdf. You get a zero if I can’t compile the pdf.
• So, ask someone else if they can run your code.

Third most important file in github repo

(After Arfon Smith)

• First: LICENSE
• Second: README.md
• Third: ???

Makefile

Learn Make if you don’t know it already.

Practical tips

• Lobby journals about their code-sharing practices.
• Lobby funders likewise.
• When reviewing articles, ask for code to be made available.
• When starting on a new project, assume code will be public at some point in the future.

Summary

• Find the selfish reasons to make your research reproducible.
• Adopt good practices to help you on your way.
• Writing code in groups can be very motivating.
• Use new tech if you want, but old tech works too.

Stephen Eglen: a brief CV

• Reader in Comptuational Neuroscience, DAMTP
• Director of MPhil in Computational Biology
• Member of Cambridge Computational Biology Institute (CCBI)
• Home page: http://sje30.github.io
• Mixing R and LaTeX (and recently markdown) since 2002…
• Advocate of Open Research

Reproducible papers

1. Eglen et al. (2014)
2. Eglen (2015)

Lessons I learnt

1. Editors loved this.
2. Reviewers engaged, edited code/figures.
3. Brittle. Paper 2 broke within 6 months!

Code review pilot

• Nature Neurosience began a pilot review project in June 2017 to check for reproducibility of key figure/table. Editorial
• We wrote some guidelines for making code reproducible. Commentary

Challenges for a reproducible paper

1. Technical aspects mostly there. Docker/Rmd/Jupyter notebooks/Zenodo/github.
2. Sustainable compute platform required. mybinder.org was a victim of its own success. beta.mybinder.org Julia example
3. Long (hours upwards) compute jobs are clearly not interactive.
4. Social challenges much harder. How to incentivise/require authors to work like this?
5. Publisher workflow should work with the author worflow (that starts before and may continue after publisher workflow finishes).
6. Will reviewers be expected to do more than read the paper?

Format-free submissions

Source